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Abstract. We study the growth rate of the summatory function of the Mobius 
function in the context of an algebraic curve over a finite field. Our work shows 
a strong resemblance to its number field counterpart, as described by Ng in 
2004. We find an expression for a bound of the summatory function, which 
becomes sharp when the zeta zeros of the curve satisfy a certain linear in- 
dependence property. Extending a result of Kowalski in 2008, we prove that 
most curves in the family of universal hyperelliptic curves satisfy this prop- 
erty. Then, we consider a certain geometric average of such bound in this 
family, using Katz and Sarnak's reformulation of the equidistribution theorem 
of Deligne. Lastly, we study an asymptotic behavior of this average as the 
family gets larger by evaluating the average values of powers of characteristic 
polynomials of random unitary symplectic matrices. 



1. Introduction 

Recall that the Mobius function /i(n) is defined for any positive integer n by 

{1 ifn=l, 
if n is not square free, 

(—1)' if n is a product of t distinct primes. 

Let M(x) be its summatory function 

M(x) := tin)- 

71 < X 

Mertens's conjecture [15] states that the inequality 

(1) |M(a;)| < >/i 

holds for all x > 1. This conjecture was disproved by Odlyzko and te Riele [TB] 
in 1985. Still, understanding the growth of M(x) remains as a subject of intensive 
investigation by many authors in analytic number theory. For example, see |16] 
and [5T] for some recent results. Relevant to us is a paper [T7] of Ng, where he 
gives certain conditional results on the growth of M(x), using the techniques of 
Rubinstein and Sarnak in [20]. In particular, Ng presents a probabilistic argument 
supporting the conjecture of Gonek, which states that the magnitude of M{x)j^fx 
grows roughly as (log log log a;) 5 / 4 . More precisely, it asserts that there exists a 
number B > such that 

(2) limsup —=- ^SeI = anc [ Hminf — — - — M( x ) _ _q 
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In this paper, we try to construct a function field analog of Ng's work and 
examine several issues that arise from this attempt. This is motivated by the 
present author's earlier paper [3] , where a function field version of Rubinstein and 
Sarnak's work is established. To describe our results in more detail, we fix some 
notations first. Let C be a nonsingular projective curve of genus g defined over a 
finite field ¥ q of characteristic p > 2 with q elements. Define the Mobius function 
Hc/¥ q (D) of C/¥ q for all effective divisors D of C in the obvious way, 

( 1 if D = 0, 

(jLc/¥ q {D) '■= \ if a prime divisor divides D with its order at least 2, 

I (—1)' if D is a sum of t distinct prime divisors. 
Also, define the summatory function 

M c/¥q (X):= Yl Vc/w q (D), 

degD<X 

for all positive integers X. 

The starting point is an asymptotic formula for Mc/v q (X) as X — > oo, which 
is given in Proposition 12.21 Roughly speaking, this formula says that Men (X) — 
0(X r ~ 1 q x / 2 ), where r is the maximum order of all inverse zeros for C/¥ q (see ((31) 
for the definition of inverse zeros and their orders.) From this, if all inverse zeros 
are simple, we deduce in Corollary 12.31 that the quantity 

M c /w (X) 
B(C/¥ q ) :=limsup g% ' 

exists as a finite number. This could be regarded as a (weak) function field analog 
of the inequality ([TJ . 

However, it is obvious that the boundedness of M^/r (X)/q x / 2 , when all inverse 
zeros are simple, stems from the fact that there are only finitely many inverse zeros 
for any given C . So, rather than studying B(C/¥ q ) for a single curve C, it would be 
interesting to find the average of B(C /¥ q ) over a family T of curves whose genus g is 
large. Instead, what we would like to do in this paper is find the average of B(C/¥ q ) 
over J- with the scalar field ¥ q growing larger. The advantage of dealing with this 
geometric average is that this set-up enables us to use the powerful tool of Katz 
and Sarnak's reformulation of the equidistribution theorem of Deligne given in |12) . 
Still, we do not quite succeed in computing the geometric average of B(C/¥ q ) but 
obtain something close to it. This is explained at a later part of this introduction. 

At this point, we need to give a definition of the Linear Independence property. 

Definition 1.1 (Linear Independence (LI)). Let 71 = ^/qc 101 , . . . ,72^ = ^/qc ie2g be 
the inverse zeros of a curve C/¥ q . We say that C satisfies the Linear Independence 
(LI) property if the set, 

{6j I < 9j < 7T with j = 1, . . . , 2g} U {tt}, 

counting the inverse zeros with multiplicity, is linearly independent over Q. 

The number field version of LI, which states that nonnegative ordinates of the 
critical zeros of the Riemann zeta function or Dirichlet i-functions are linearly 
independent over Q, plays a key role in the aforementioned work |T7| of Ng, as 
well as in the work of Rubinstein and Sarnak on the prime number race in [20] . 
Note that LI is called the Grand Simplicity Hypothesis in [20] and [3]. The fact 
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that such a property on zeta zeros has a consequence in understanding the ex- 
act growth of M(x) had already been made clear by Ingham in j9], well before 
Odlyzko and te Riele disproved Mertens's conjecture. Ingham proved in 1942 that 
lim supj-^oo M(x)/ \fx — oo if LI holds true for the Riemann zeta function. 

Unfortunately, there is currently very little direct theoretical or numerical evi- 
dence to support the number field version of LI. However, in the function field case, 
things are better understood. There are known examples (see [3], [5] and [11]) where 
LI can be confirmed positively and, in some other cases, negatively. Moreover, the 
work |11) of Kowalski shows that most curves in a certain one-parameter family 
of hyperelliptic curves satisfy LI. For more background and precise statements of 
Kowalski's results, the readers are referred to [TT] and Remark 13.21 of this paper. 

The importance of LI in our work comes from Theorem 12. 51 which states that, 
if C satisfies LI, then the bound 

7 

of B(C/¥ g ) we found in Corollary becomes sharp, that is, B(C/W q ) = D(C/¥ q ) 
under LI. It turns out that Kowalski's argument in [TT] can be easily extended to 
prove that most curves in the family T-L^g+l of hyperelliptic curves of genus g satisfy 
LI (Theorem 13. lj) . As a consequence, we have that B(C/¥ q ) = D(C/¥ q ) for most 
curves C/¥ q in %2g+i- This is why we choose to work with T := %2g+i in this 
paper. 

The next step is to use Deligne's equidistribution theorem to find the geometric 
average of D{C/¥ q ). We show in Theorem 13.31 that a certain truncated version of 
the geometric average of D(C/¥ q ) is equal to the integral 

Ag) ■■= I ^(C/)d M Haar(C/). 
JUSp(2g,C) 

Here, d/iHaar is the unique probability Haar measure on the unitary symplectic 
group USp(2g,C). The function if is defined in the equations ([T9]) and (|20| . to 
which we refer the readers for its definition. But, we note here that this situation is 
similar to the number field case in |17] where the discrete negative moments J-k(T) 
of C'( s ) play an important role in understanding the growth of M(x). 

In <21 which can be read independently of other parts of this paper, we study 
the integral 1(g), especially its asymptotic behavior as g — > oo. A key result 
here is Theorem 14.11 which finds asymptotic expression of the average value of 
positive powers of the characteristic polynomials of random unitary symplectic 
matrices. Main tool is a recent formula in [5] by Deift, Its, and Krasovsky on 
Hankel's determinants with singular weight functions. 

Our result seems to suggest that the geometric average of B(C/¥ q ) over H^g+i 
is given by the asymptotic formula in (|38|) . However, there are, at least, two major 
issues we cannot resolve in this paper. First is the extent of the possible failure 
of LI in W2g+i- Without LI, it may happen that B(C/¥ q ) < D[C/¥ q ), therefore, 
the geometric average of D(C/¥ q ) might potentially overestimate that of B(C/¥ q ). 
Even though the set of conjugacy classes in USp(2g,C) whose eigenvalues have no 
(nontrivial multiplicative) relations form a measure zero subset with respect to the 
Haar measure, it is still dense in USp(2<7, C), and it is unclear if one could utilize the 
equidistribution theorem to control the difference between the averages of B(C/¥ q ) 
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and D(C/¥ q ). The second issue is that we cannot (yet) finish the proof of the 
asymptotic formula (|38]) of X(g). To do so, it seems that we need to have a finer 
control on the error term in the formula of Deift, Its, Krasovsky. For more detailed 
explanation on this point, see $4] especially, Remark 14.21 



Acknowledgements. The author is greatly indebted to Emmanuel Kowalski for 
helpful discussion and to Jinho Baik, who brought to the author's attention the work 
of Deift, Its and Krasovsky [6: and explained its usefulness in proving Theorem l4.ll 
Also, the author is thankful to Peter Humphries, who found an error in Proposition 
12.21 in an earlier version of this paper, in addition to offering many other helpful 
remarks. 

2. Asymptotic formula and the Linear Independent property 

Throughout this paper, we write #A for the cardinality of for a finite set A. 
We fix a power q of an odd prime p > 2 and we denote by ¥ q a finite field with q 
elements. For each n > 1, we have a unique extension ¥ q ™ (inside a chosen algebraic 
closure of ¥ q ) over ¥ q of degree n. For a nonsingular projective of curve C over F 9 
of genus g, the zeta function Zc/¥ q {u) of C over ¥ q is defined by 



z c/¥ q {u) :=exp ^ 



which is initially defined as a formal power series in u with rational coefficients. It 
is known from the Riemann hypothesis for curves over finite fields that 

P C/¥ q (u) 



(3) z c/w q (u) 



{l-u)(l-quy 



where Pen (u) is a polynomial in u with integer coefficients of degree 2g, which 
factorizes as 

(4) P c/¥q (u) = l[(l~ lj u) 

for some complex numbers "fj with \ = yjq for all j = 1,. . . , 2g. These numbers 
jj are called the inverse zeros of C . By an order of an inverse zero 7, we mean the 
multiplicity of 7 _1 as a root of P c / ¥q (u). If 7 is of order one, we will say that the 
inverse zero 7 is simple. 

2.1. Asymptotic formula of M c / ¥q (X). Define Z^u) to be the following Dirich- 
let series (in u) associated with fi c / ¥q (D), together with the change of variable 
u := q~ s , 

(5) ZM ■= E - t 

L>>0 N=0 

Here, AfD is the absolute norm of the divisor D and c^(N) :— Yldeg(D)=N Mc/F, 
From the Euler product expression of Z^u), it is easy to show (see Chapter 1 of 
|19) or follow the same argument as in the number field case) that 

1 (1 - u)(l - qu) 



(6) Z^u 



Z C/¥ q (u) P C /¥ q (u) 
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From ([5]) and the definition of c^(JV), we have that 



(7) 



ft I, 



c/w. 



N<X 



Therefore, the crucial step in finding the asymptotic formula of ftic/r (X) is to 
estimate the coefficients c M (iV). 

First, we consider the easiest case when C is of genus 0. In this case, Pc/w ( u ) = 1 
and 

Z^[u) = (1 - u)(l - qu) = 1 - (q + l)u + qu 2 . 

So, e^N) = for all N > 3 and c^N) = l,-(q+l),q if N = 0,1,2 respectively. 
This easily determines the values of Mc/t q {X) for all X. In particular, we obtain 
the trivial bound of \M c / ¥ (X)\ < q for all X. 

Next, we consider the general case for arbitrary genus g. Let G\ is a circular path 
in the complex plane of radius 1 centered at the origin, oriented counterclockwise. 
We calculate the integral 



1 

2?ri 



„N+1 



du 



using Cauchy's theorem. 

First, we note that the above integral can be easily bounded independently of 
N. From jBJ and 



(«) 



1 

2ni 



Z n( u ) 
,,N+i 



du 



< 



1 

2^ 



Z^u) 



,N+1 



\du\ < 



2(1 + 9) 
{1-^/qfB- 



Next, we see from ([5]), ©, and d3| that the function Z,i(u)/u N+1 has residues at 
u = and u = 7 _1 for all inverse zeros 7. The series expression ([5]) implies that the 
residue of Z /1 (u)/u N+1 at u = is c^(N). Therefore, if we define Rc/w q {N, 7) to 
be the residue of Z )Ji (u)/u N+1 at u = 7 _1 for any inverse zero 7, Cauchy's theorem 
yields 

2(1 + «) 



c„(JV)+X;flc/F t (iV,7) 



< 



2s ' 



By letting AT — > 00, we get 
(9) c M (7V) 



^i? c/F? (iV,7) + 0(l). 



So, in order to obtain an asymptotic formula for Mc/w q (X), we will need to calculate 
the residues Rq/f (^,7)- We do this by finding the Laurent series expansion of 
Z^u)/u N+1 = l/{Z c/¥q {u)u N + 1 ) directly. 
From binomial theorem, 



(10) 



-(JV+l) 



fc=0 



iV + k 
k 



7>-7~T- 



Let r be the order of 7. Then the power series expansion of Zc/w q ( u ) at w = 7 
starts with 



Z C/F q (u) 



{u- 1 - i y + 
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Here Zc/r q ^ (u) is the r-th derivative of Zc/v q (u) (with respect to the variable u). 
Then, the Laurent series expansion of 1/Zc/v (u) at tt = 7 _1 begins with 

Z C /¥ q (u) Z c/r } r) ^- 1 ) 

Therefore, the residue Rc/¥ q {X,^) is obtained by multiplying the two series (p~0|) 
and (fTTj) and extracting the coefficient of (u — 7~ 1 )~ 1 . To be precise, 



R C/rq (N, 1 ) = - ^__yv + r ( _ iri( j 7 r-i + ... 



^c/F 9 (r) (7- 1 ) 



. 7 iV + l ( _ 1) r-l^_^ 7 r-l + 



(12) =2 -N r ~ 1 + ..., 

Zc/w^h- 1 ) 

where all the suppressed terms are polynomials in N of degree r — 2 or less. 
If we sum the equation (|12p over iV = 1, • • • , X, we get 



(13) t *c /Vq (N, 7 ) = / ( ^r 1 n ^ L T Xr " V + ^V), 

AT=1 Z C/F 9 Vf ) 7 1 

as X — > oo . This is an immediate consequence of Lemma 12.11 which can be proved 
by partial summation [51 Theorem 4.2], as outlined in [3J Lemma 2.2]. So, we omit 
the proof. 

Lemma 2.1. Let /3 be a complex number with \/3\ > 1 and k be a nonnegative 
integer. 



lim 

X^oo X 



— IV N k B N ) = JL- 



Denote by #(7) the argument of the complex number 7, so that 7 = ^Jqe l9 ^\ 
Then, (fT3"j) becomes 

(14) V-xB Y Rc/w (N, 7) = ( 7! V ^-e iX ^> + o(l). 

1 ' * r -\ Xl2 ^ C/ qK Z c/¥q ^ (7-1)7-1 W 

Now, the equations (HJ), ©, (fT4")l together yield the estimate of Mp/f (X) in Propo- 
sition [~~ 



Proposition 2.2. For an inverse zero 7, Zet #(7) &e i/ie argument of 7, so </ia< 
7 = ^fqc 10 ^. Also, let r be the maximum order among all the inverse zeros 7 of 
z c/¥ q {u), that is, 

r = max{ order of 7j}|=i 

Then, as X — > 00, 

^cW^) = v (-7) r ^ 7 ixeh) . fn 
^"V/ 2 ord ^ =r ^«(7- 1 )7-l U - 
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In particular, if all inverse zeros {7j} 7 -jLi are simple, the 



M c/rq (X) 



2g 



7j 



7 X/2 



^ ^c/F 9 '(7j J ) 7j 



1 



0(1). 



Corollary 2.3. TTie notations are as above. 



E 

ord7— r 



7 r 



^c/F g W (7- 1 )7 



The next theorem follows from the adaptation of Rubinstein and Sarnak's argu- 
ment in [20] for the function field setting and is identical to that of Theorem 3.2 in 
[3], so we omit its proof. 



Theorem 2.4. The notations are the same as in Proposition \ 2.SX The function 
Mc/¥ q (X)/(X r ~ 1 q x / 2 ) has a limiting distribution [i on K, that is, 



lim . 



X=l 



M C /v q (X) 

X r-lqX/2 



f{x) d/i(x) 



for all bounded continuous function f on 



2.2. Application of the Linear Independence property. Suppose that the 
curve C satisfies the Linear Independence property (Definition [TTT]). One immediate 
consequence of LI is that all of inverse zeros of C are simple, therefore, the formula 
in Proposition 12.21 becomes 

7j 7j 



(15) 



M, 



C/F, 
jX/2 



(X) 



E 

i=i 

where u>{jj) is the argument 
"(7j) : 



arg 



7? 



co B (w( 7i )+-X'(9(7 i )) + (l), 



7j 



^c/f/(7 j -- 1 )7 j --1. 



Another consequence of LI is that the measure fi whose existence is stated in 
Thcorcm l2.4l is absolutely continuous and we can write down the Fourier transform 
ft explicitly 



(16) 




^c/f/(7 j -- 1 )7,'-1 



where Jo(z) is the Bessel function of the first kind. Again, (the adaptation of) 
Rubinstein and Sarnak's proof can be used to establish this, so we omit the proof 
of this formula. However, this fact allows us to prove that the sum of magnitudes of 
the oscillating terms in the right side of (|15|) , which provides a bound of the left side 
of (|15[) . does indeed give the sharp bound as X — > oo. This is the key observation 
which we will use in the next sections, and we summarize it as a theorem below. 



Theorem 2.5. If C satisfies LI, then we have 



lim sup 

X— >oo 



7 X/2 



2fl 

E 



7j 



7j 



z c/¥ q (7j *) 7j 
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3. Universal families of hyperelliptic curves 

We define H2g+i to be the space of monic polynomials of degree 2g + 1 with 
distinct roots (see (10.1.18.1) of [H]). One can think of W2g+i as an open subvariety 
of the afhne scheme A 2g+1 over Z. In particular, for each n > 1, H2g+i(V q n) is the 
set of monic polynomials f(x) — ao + a\x + • • • + a2 g x 2g + x 2g+1 with coefficients 
a; in ¥ q n where the discriminant of this polynomial is nonzero. Therefore, each 
/ £ H2g+i (F g » ) defines a hyperelliptic curve Cf of genus g over F 9 >. , the nonsingular 
projective model of the plane curve defined by the equation y 2 = f(x). At this 
point, it will be convenient to introduce a terminology from [4]. We will say that 
most points of r H.2g+\ have the property D — {D n }^ =1 if 

#{/ £ 7k fl +i(Fgn) | C f satisfies J n } = 

™ #H 2s+ l(F q n) 

3.1. LI for most curves in W2g+i- 

Theorem 3.1 (Chavdarov [4j, Kowalski [IT]). For fixed q and g, we have that 
j. #{/ £ H 2g+ i(F g .) | C/ satofies LI. } = 

n^o #^ 2ff+1 (F g „) 

/n other words, most points of %2g+i satisfy LI. 

Proof. Let / and C/ be as above. Then one can show that, for most points of T^g+i, 
the sum of inverse zeros of Cf is nonzero. This directly follows from Deligne's 
equidistribution theorem (Theorem 10.8.2 of [H]) because the set of conjugacy 
classes with zero trace forms a measure zero subset of the space of conjugacy classes 
USp(2g, C)# of USp(2g,C), with respect to the (direct image of) Haar measure. 

The second step is to apply Chavdarov's theorem (Theorem 2.3 of j4]) which says 
that, for most points of %2g+ii the Galois group of the splitting field of Pc f (u) is 
as large as possible, that is, the Galois group is isomorphic to the Weyl group W2 g 
corresponding to the symplectic group Sp(2g). To apply Chavdarov's theorem, we 
need to ensure that the mod-^ geometric monodromy group of H.2g+\ is Sp(2.g, F^) 
for all large t. But this result had been previously obtained by J. K. Yu, which is 
unpublished. More recently, Hall in [7] and, independently, Achter and Pries in p], 
proved this result. 

The last step is now to follow Kowalski's argument in §3 of [11] , His proof of 
Proposition 1.1 in §3 of [IT] can be applied to %2 ff +i without any change to show 
that, if the sum of inverse zeros of C is nonzero and the Galois group of Pc f (u) is 
as large as possible, then C satisfies LI. This concludes the proof of our theorem 
that most elements of H2g+i satisfy LI. [1] □ 

Remark 3.2. The aforementioned Chavdarov's theorem has a quantitatively re- 
fined version, which was first proved by Kowalski. See Theorems 6.1 and 6.2 in 
[lOj , where Kowalski gives a quantitative bound on the number of curves in a fam- 
ily whose zeta functions are either reducible or have splitting fields with strictly 
smaller Galois groups than the maximum possible one. And, using this result, 
Kowalski derives a bound of the number of curves which don't satisfy LI in the 
following one-parameter family of hyperelliptic curves 

C t :y 2 = f(x)(x-t), 
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where f(x) is a monic irreducible polynomial with coefficients in Z whose discrim- 
inant is not divisible by p (Proposition 1.1 of [H]). In fact, if we assume that 
p > 2g + 1, then Theorem 6.1 (ii) of [TU] is directly applicable to the family "H 2s +i, 
and we can deduce from it that the number N (%2g+i{¥ q )) of curves in H2g+i{¥ q ) 
such that Pc f {u) is either reducible or has splitting field smaller than W<z g satisfies, 
as q — >• oo, 

N(n 2g+ l(¥ q ))^q 29 -'(logq), 

for 7 := l/(10.g 2 + 6,g + 8). Therefore, the same bound above applies to the number 
of curves that don't satisfy LI. 

3.2. Average over the family. Let C be a nonsingular projective curve over a 
finite field F of characteristic p > 2. As in let r be the maximum order of all 
inverse zeros of C/F. Define 

M C/¥ (X) 
B(C ¥) := limsup ° /¥ ) ' - 

Further, we let 



ord(7j— r 



7 r 



Then, Corollary 12.31 and Theorem 12.51 can be summarized by saying that 

(18) B(C/¥) < D{C/¥), 

and that the equality holds true if C satisfies LI. In this subsection, we investigate 
a relationship between an average value of D(Cf,¥ q n) for all / e "%2 5 +i (F2 9 +i) 
and a certain integral over the unitary symplectic group USp(2<7, C). 

To describe this relationship, we start by defining the characteristic polynomial 
Zu{&). (A typographical note: in the literature, this characteristic polynomial is 
denoted by ZjjiO) or Z(U,9). But, we use a calligraphic font in this paper to 
distinguish it from the zeta function Zc^JyU) of a curve C.) Let TV be a positive 
integer. For a 27V x 27V unitary matrix U and a real number 6, wc define the 
function Zjj(6) by 

27V 

(19) Zu(0) := det(I - Ue- W ) = J[ (1 - e^"^), 

m— 1 

where e lSl , . . . , e i02N are the eigenvalues of U. When U has no repeated eigenvalues, 
we define 

2N 

(20) <p(U) = V 



j 



\\Zv' 



Note that tp(U) depends only on the conjugacy class of U. In this paper, we will be 
mostly interested in <p(U) for a unitary symplectic matrix U & USp(2g,C). Then, 
the function <p(U) is continuous and well-defined outside the measure zero subset 
where U has a repeated eigenvalue. 

Next, for a curve Cf over F g n, we recall that there exists a conjugacy class 
i?(C//F g n) in the set of all conjugacy classes USp(2 ff ,C) # of USp(2 5 ,C). This is 
called the unitarized Frobenius conjugacy class attached to C//F g n. For its defini- 
tion, the readers are referred to Chapters 9 and 10 (especially §9.2 and §§10.7.2) 
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of [12] , In this paper, it will be sufficient to say that this is the unique conjugacy 
class with the property that 

(21) P Cf/Vqn (u) = det(l -uq n ^(C f /¥ qn )). 

Finally, we define a truncated version of D(Cf,¥ q n) using (f(U) above. Fix a 
positive number T > 0. Then, 



(22) D T (C f /¥ q r, 



£>(C/,F,») ifp(0(C>/F,«))<T, 
otherwise. 



Note that the second case in the above definition is used when either ip({)(Cf /¥ q n)) > 
T or i9(Cf/¥ q n.) has a repeated eigenvalue. The truncated average over %2g+i(^ V) 
is defined by 

(23) £ T (H 2g+1 (V)) := — 1 Y, DT ( C f/^)- 

#% 9+ i(F 9 ,) /eW2B+i(Fgii) 

We are ready to state the main theorem of this section. 
Theorem 3.3. Notations are as above. 

lim \\^D T (n 2g+1 {¥ qn ))=[ <p(U)d m ^{U). 

T-K» n-K» ' JUSp(2g,C) 

Here, d/ijjaar is the unique probability Haar measure on USp(2g, C). 
Using ([TO]) and (|2"Tj) . we can easily deduce that 

(24) Z HCf/¥qn) {d) = P Cf/¥qn {{q n / 2 d e )- l l 

for any real 8. Assume that C/ has only simple inverse zeros and write jj = q n / 2 e l9j . 
Then, we differentiate the equation (|24|) to obtain 

/ 25 x 7j lj_ = 1 - (g"/7j) 

Further, assume that (p($(Cf /¥ q n)) < T. Then, we sum (|25|) over j = l,...,2g, 
and, by setting r = 1 in (|17p . this yields 



1 - (9 n /7i) 



(26) =£/ 1 1 ^ 



where A(f,j) is a constant with j)\ < 1. Now, we compute D (T^g+i {¥ q n) 
from its definition (|23|) by adding (|26|) over all / G W.2g+i for those / with ^(i?(C/ /F g n )) < 
T. As a result, 

(27) If (H 2fl+1 (F g .) = * £ E 



#% s+ i(F 9 „) ^ (fl(C//F5re)) < Tj=1 |3* (c ,/F t »)'(0i)| 
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The next step is to pass n — > oo in (|27[) and to apply Deligne's equidistribution 
theorem, Theorem 10.8.2 of [12]. The right side of the first line of (f27|) then becomes 



I 29 I 



" °° ^2 ff+ l(F 9 ») V( ^ (C ^ )) < T ^ l^/F.n) (^)l 



tp<T 



The second line of ([27| converges to zero as n — > oo due to the g™/ 2 term in the 
denominator and the convergence of the first line of (|27p . In other words, we proved 
that 

lim D T CH 2g +l(¥ q n)) = f ^(C/)d MH aar(C/). 
™^°° ' J V <T 

The proof of Theorem 13.31 is now completed by letting T — > oo. 

4. Averages of characteristic polynomials on unitary symplectic 

GROUPS 

Recall that, for a 27V x 27V unitary matrix U and a real number 0, the function 
Zu{&) was defined in (|19p by 

2N 

Zu(6) := det(7 - t/c- ie ) = JJ (1 e^"^), 

m— 1 

where e lSl , . . . , e 102N are the eigenvalues of U. Also, the function <p(U) is defined in 
® by 

whenever U has no repeated eigenvalues. When U is an element in USp(27V, C), its 
eigenangles 9\, . . . ,92N come in complex conjugate pairs. And we will enumerate 
them in the way that < 6j < ir for j = 1, . . . , TV and 0n+i = —9\, . . . , 2 n = —Qn- 
The main theorem of this section is Theorem 14. 11 where we give an asymptotic for- 
mula of the 2s-th moments, for any positive real number s, of Zjj(ff) in USp(27V, C) 
using a recent work [5] of Deift, Its, and Krasovsky. 

Theorem 4.1 (cf. Theorem 5 in |13|). Fix a real number s > (not necessarily an 
integer) and 9 with < 9 < n. As TV — > oo, 

/USp(2jV,C) U(i + ZS) 

Here, G{z) is the Barnes G-function. 



Jul 



Proof. During the proof, we use the notation 

(28) (\Zu(9)\ 2s ) V s P (2N,€) ■= I \Zu(9)\ 2s d(i KaaI (U). 

JUSp(2N,C) 

We first rewrite 

N N 

(29) \Zu(9)\ = Yl |1 e i(9 ^ 9) ||l - e i( ^ +9) | = 2 N J[ \ cos9 3 - cos9\. 

This can be done by applying the following straightforward trigonometric identity 



(30) 



1 _ e Wi-0k) 



1 



= 2 1 cos 9j — cos 9k | 
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to (|T9j) . To integrate (|29|) . we use the Weyl integration formula, which describes 
the Haar measures on classical matrix groups explicitly in terms of eigenangles. 
The version we use here is (5.0.4) of 12 , which we recall below. Dehne a measure 
At(USp(2A0) on [0,<k] n by 



,jv^ N N 



2 

(31) d M (USp(2iV)) := ]J (cos^ -cos0 fe ) 2 []sin 2 ^[] d ^' 

l<j<k<N 3 = 1 3 = 1 

where d6\, . . . ,d0N are the usual Lebesgue measure on the set [0,7r]. Then, the 
Weyl integration formula says that, for a bounded, Borel measurable ]R-valued 
central functions g on USp(2iV, C), we have 

(32) f g(U)d maar (U)= [ g(9 1 ,...,0 N )dfx(VSv(2N)). 

J\JSp(2N,C) J[0 : iv] N 

Here, g is the function on [0,^]^ defined by the property 

g{0i,...,0 N )=g(U), 

whenever 8\, . . . , 9n,—9i, . . . , — 9m are the eigenangles of U G USp(27V, C). Now, 
from J28J, ([29]), 43]} and <j32j) , we have 



nN 2 +2sN r 

(\Zu(0)\ 2S )lJS P (2N,C)= M , N / [J (COS^-COS^) 2 

JV 

x Yl I cos 6j - cos 6\ 2s sin 2 6, d6j . 

i=i 

Note that the expression IIi<j<fc<Af ( cos @k~ cos #.,■) is the same as the Vandermonde 
determinant A(cos#i, • • • , cos9j) where 

A(xi, . . . , a; at) := det(aif'~ 1 )i<i, i <jv 

Set y = cos6. Then, clearly, < y < 1. Also, we use the change of variables 
xj = cos 8j to obtain 

2N' 2 +2sN r N 

(33) (\Zu(0)\ 2a )vsp(2N,C) = M , N / A(x u ...,x N ) 2 T\wy(xj)dxj. 

Here, the weight function w y (x) is defined by 

w y (x) := \x — y\ 2s \/l — x 2 . 
Now, we will use the following Andreief's identity, 

1 f N 

(34) — / det[f j (x k )] 1 <j !k < N det[gj(x k )} 1 < jik < N Y[w(xj)dx j = 

■ J* n j=l 



dct / fj(x)gk(x) w(x) dx 

Jx J l<j,fc<JV 

for any interval X in M. Setting fj(x) = g 3 {x) = x^ 1 and X = [—1,1] in the 
Andreief's identity, we can rewrite the integral (|33l) as 

nN 2 +2sN 

(35) (\Zu(9)\ 2s > USp(2J v,c) = „ N det 

0<j,fe<JV-l 



x j+k w y (x) dx 

l 
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The determinant in (|35|) is, so-called, a determinant of Hankcl's type [22] with 
weight w y (x). And what we need is an asymptotic expression of Hankel's determi- 
nant when the weight function is not differentiable. We use a recent result in [5] 
of Deift, Its, and Krasovsky, which can be applied to much more general weight 
functions than our w y (x). In particular, Theorem 1.20 of 6 , with the following 
parameters, 





a 


:T J 


'A 


= 1, 






V = 0, 6± = 1, < 


ax 




Ai 


= y, 


and / 


% = for all i 






= 1/4, 


\2 


= -i, 







gives 
(36) det 



x }+k w y (x) dx 



0<j,k<N-l 



= 4-(«w+f +f +A) (27T) 1 / 2 AT- 

G(l + .s) 2 
G(l + 2s) 



as TV — > oo. Further simplification yields 
-i 



det 



x J+k w y (x) dx 



0<j,k<N-l 



2N 2 +2sN 



2-^ 2 (l- y 2 )-^^/ 2 §|i±|l y (l + (l)). 



Combining this with (|35j) (and remembering y = cos 9), we finish the proof of the 
theorem. □ 



In view of Theorem l3.3[ it would be desirable to obtain an asymptotic expression 



of 
(37) 



1{N) 



USp(2JV,C) 



<P(U) d AtH aar(f/). 



as N — > oo, because this would be then thought of as a function field analog (for 
the family T-^g+i) of the (log log log x) 5 / 4 term in ([2]). Using Theorem 14. II (for the 
case s — 1/2), we present some evidence in support of the formula 

(38) I(N)~ V2G(i) 2 S(|,i)7V 1 / 4 . 

Here, G(z) is the Barnes G-function and B(x,y) is the beta function 



B(x,y) 



[ t x - l {l-t) y - l &t, 
Jo 



The rest of the paper is now devoted to presenting the argument in support of 
the formula ([58)1 . Our computation closely follows the strategy used by Hughes, 
Keating, and O'Connell in the proof of Theorem 1.2 in [3J. 
A straightforward differentiation of Zjj(8) in (fT9t gives 



n 



(39) 



\Zu'{Bj)\ = |1 - e 2i ^| JJ |1 - e^-^Wl - e^'+H 



fc=i 
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for j = 1, . . . , N. Using (|30|) and another easy trigonometric identity 

|l-e 2i ^| =2|sin^|, 
one easily deduces from (|39|) that 



N 



\Zu'(0j)\ = 2 N \ sm9j\ Y[ | cos^ - cos0 fe | 



k=l 



Also, obviously, \Zu(0 N+j )\ = = \^u(0j)\ for j = 1, . . . , AT. Hence, 

27V AT W 

(40) p(L0 = £ \Z u '{9 m )\- 1 = 2 1 " iV ^| sin^-r 1 [] | cost% - cos^T 1 . 

i=i 



m— 1 



fe=l 



To integrate (fi(U) over USp(27V, C), we use the Weyl integration formula again. 
That is, from PO)) . (|5T |) and (|52"1). 



USp(2AT,C) 

oA^-AT+l 



N\TT 



lfi{U) d^Haar(C^) 
AT 

^Isin^l- 1 Y[ I cos - cos 9 k T 1 



AT 



A* 



j=l 



fc=l 



A' 



A ? 



x jQ (cos 6j - cos e k ) 2 J| sin 2 ^ d0j . 

l<j<k<N j = l j = l 



The expression in the square bracket inside of the above integral is symmetric in 
Oj's, therefore, we can replace the summation on j by N times any single summand, 
say, the j — N term. Then we proceed 



(41) 1(N) = 



2N 2 -N+1 



2N 2 -N+1 



N-l 

sin0Ar| _1 JJ | cos^at — cos6*fc| 
fe=i 

N N 

x yi ( c ° s °3 - c ° s °k ) 2 n sin2 Q i n d °3 

l<j<k<N j=l j=l 

N-l 

| sin 9n I Y\_ I cos6*jv — cos0fe| 



k=l 



N-l 



N 



x Yl ( cos 9 i - cos 6» fe ) 2 Y[ sm2 °j Yl d ^ ' 

l<j<k<N-l j=l j=l 

Again, (|30)) can be used to rewrite the expression in the square bracket after the 
last equality in (|4T]) as 



N-l 



N-l 



s'm8 N \ Y\ I cos 9 N - cos 9 k \ = | sin 9 N \ Y\ ^ |l - e 1 



(0k — Qn) 



k=l 



k=l 



1 _ e K0k+e„ 



>1— JV| 



sin 6>jv | |-Z"c^(6'jv) | , 
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where U is an element in USp(2(iV — 1)) whose eigenangles are ±#i, . . . , ±#jv-i- 
So, we continue from (|4Tj) to get 



o(iV-l) 2 ( 

(42) J(iV) = — — — / 2|sin0 JV ||Z [ ,(0 J v) 
(TV - l)!vr JV 7[o,ir]Jv 



N-l N 

x ]Q (cosfy - cos6» fc ) 2 n sin 2 9 3 JJ d0j 

l<3<fe<AT-l 3=1 3=1 

2 



/ sm9 N [ [ \Zu(d N )\d maiiI (U))de N , 

J\0,tv] \JvSv(2(N-l)) I 



n J[0,tt] \JUSp(2(Af-l)) 

where the last equality is again from the Weyl integration formula (f3Tj) and (|32|) . 
applied to USp(2(iV— 1)). The integral in the last line of (|42|) within the parenthesis 
is precisely the integral (\Zu(9n)\ 2s )uSp(2(iv-i)), with s = 1/2, whose asymptotic 
expression is found in Theorem 14.11 Therefore, after some simplification, we find 
that 

(43) 1(N) = V2G(^) 2 N^ 4 f \smefl\l + o(l)) d9, 

Jo 

as N —> oo. 

Remark 4.2. In order to prove the asymptotic expression (|38|) . we let N oo 
in (|43p . Then, if we can exchange the limit and the integral, then the integral of 
(sin^) 1 / 4 is expressed in terms of the beta function and then this would finish the 
proof of (f3"5| . The key step here, therefore, is to estimate the size of o(l)-term in 
(|43|) with respect to 9. 

This error term comes from the formula of Deift, Its, and Krasovsky, quoted in 
(f3l>)) . See Remark 1.6 of [B] for some general discussion on the size of their error 
term. Let e^(S) be the o(l)-term in (|4"3")l . If one can show that the expression 
(su\9) l / i eN{9) is bounded by a function in L 1 ([0,7r]) independently of N, then the 
dominated convergence theorem can be used to justify the exchange of limit and 
integral. 

In fact, we can show that cn{9) does not tend to zero uniformly in 9 as follows. 
Define 



f N {9) := / \Zu{9)\d maal {U). 

JUSp(2N,C) 

Then, it is known that 

MO) ~ N, 

from a result in |14) . (Or, alternatively, one can use the same formula of Deift, 
Its, and Krasovsky and proceed exactly as in the proof of Theorem 14.11 ) If we 
assume that the error term in Theorem 14. II is bounded uniformly in 9, we can pick 
N large enough, so that /n(9) is about (a constant times) iV 1 / 4 (sin6') _3 / 4 , for all 
9 close to 0. Now, if we now choose 9 in the range < 9 < l/N 2 , then this gives a 
contradiction to the continuity of /n{9) at 9 = 0. So, this implies that ejv(#) does 
not tend to uniformly in 9 as N — > oo. Therefore, further investigation on cn(9) 
is warranted to justify the exchange of limit and integral in (|43|) . 
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